What Your Reference Tracks Actually Tell You